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ABSTRACT 

The utility of Just-In-Time-Teaching (JITT) is compared across course topics and groups of 
students not receiving JITT exercises in class. JITT feedback incorporated various active learning 
exercises based on students’ performance on online homework problems from Sapling Learning. 
With over 200 students in two sections participating in the sophomore level introductory thermo¬ 
dynamics course, student performance was evaluated on in-class quiz and exam problems. Student 
performance covering a specific course topic subsequently reinforced by a JITT exercise was found 
to be measurably higher (>9%) than a control group who did not receive the JITT review. Overall, 
comparing student achievement across five topics covered by JITT compared with five different 
course topics not covered by JITT did show small positive outcomes. The improvements after us¬ 
ing JITT exercises included a higher median and 3 rd quartile scores, so JITT may be beneficial to 
mid-performing students. Overall, student feedback on the JITT reviews showed that over 85% of 
the students thought the JITT exercises were helpful for engagement and a good use of class time. 

Key words: Just-in-time-teaching, thermodynamics, Sapling Learning 


INTRODUCTION 

Active learning encompasses a large set of pedagogies that engage students in the class¬ 
room 1_3 . Recently, the integration of technology with active learning has provided educators with 
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quantifiable data sets on the effectiveness of different active learning techniques. For example, 
students responding via electronic clickers improved in their conceptual understanding in sciences 
and engineering 2 - 4 6 . In this work, the technology is an outside of the classroom online homework 
platform that provides the instructors data to complete just-in-time-teaching exercises related to 
difficulties in the homework. 

Just-In-Time-Teaching (JITT) defines a set of active learning pedagogies 7>8 . JITT can be used to 
clarify new concepts and eliminate misconceptions by linking work students do prior to class with 
classroom exercises, explanations, and lectures. Developed to engage students in class and provide 
real time feedback 8 , the use of JITT has spread to many diverse disciplines; however, few engineer¬ 
ing applications have been published 7>9 . 

In general, JITT builds cognitive and metacognitive skills using a number of established learning 
principles, which will be briefly summarized here. JITT provides a flexible, but structured mechanism 
for the students to complete work, usually outside of class, that provides the instructor feedback 
on learning gaps. Then the instructor can use class time to provide feedback through additional 
student-centered activities. A number of examples tying research on learning 1012 to the suite of JITT 
techniques can be found in the books and journals 7 ' 9;l3 ' 15 . For example, presenting the aggregate 
student responses to the class, a misconception may be identified for the majority of the class. 
Then an exercise can be completed to repair the misconception while simultaneously building the 
students’ metacognitive skills 16 ' 18 . Overall, many new technologies can provide instructors with real 
time data to identify learning gaps and instruction can then focus on building a more complete 
understanding of the subject, which has been applied across a range of teaching styles, courses, 
and learning environments 7 

In order to deliver just-in-time teaching, data must be collected to measure the understanding 
of the students on recent material. While a growing body of literature involves concept ques¬ 
tions and clicker-based responses 2 - 6 ’ 16 - 19 - 23 , online homework provided the quantitative assess¬ 
ment in this study 24>25 . The authors worked with Sapling Learning© to implement over 150 new 
problems for the introductory thermodynamics course. While numerous online homework tools 
exist for basic science courses, such as freshman chemistry or physics (see any major publisher), 
the availability for engineering courses is much more limited. Having experience teaching the 
thermodynamics course over many years, the authors had a collection of problems that they 
had authored and uploaded using Sapling’s authoring environment. The authoring process is 
beyond the scope of this work, but the salient features of the online homework environment 
will be detailed next. 

Our implementation of JITT has students completing online homework using Sapling Learning. 
Online problems are graded as they are completed so that the student gets immediate feedback 
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and the instructors can track the overall class success and/or problem areas before the class meets. 
Instructors reviewed assignments approximately 2 hours prior to class and developed problems 
to be presented as in-class exercises or mini-lectures. Students, working in groups of 2 or 3, then 
worked the in-class problem followed by an instructor explanation and solution. Key concepts were 
emphasized and explained, and common misconceptions and errors were noted and corrected. This 
paper discusses using real time online homework data with JITT to improve student achievement 
on quizzes and exams. 


METHODOLOGY 

Several features of online homework provide an advantage over traditional paper homework from 
a textbook (Figure 1). First, the problems contain rolling numbers or choices so each student has a 
slightly different set of problems. Rolling the numbers helps to avoid the usage of “textbook” solution 
manuals and their availability to students has been discussed in recent years 9>26 . Rolling numbers are 
easy to implement for algorithm problems, but changing numeric values is more challenging when 
physical properties are needed. Implementing at least three sets of numbers for problems involving 
tabular data (i.e., steam tables) provided enough variation with the goal of discouraging students 
from copying answers from other students. Having rolling numbers on exams has led to a number 
of academic dishonesty cases in the authors’ experience. By having students work on their own 
numerical solution, the focus is more on problem solving and learning than on getting the correct 
answer, which adheres to the authors’ teaching philosophy. 

Next, the problems contain hints to help the students complete the problems without synchro¬ 
nous intervention by the instructor(s). Hints are normally viewed after an unsuccessful attempt to 
answer the question. While unsuccessful attempts lead to a small penalty (10% per attempt in our 
implementation), multiple attempts encourage persistence and likely improves learning 25 . Finally, the 
problems are automatically graded so both the student and instructor receive immediate feedback. 
Figure 1 shows a screen shot of a typical Sapling problem and illustrates which of the numbers are 
changed for the individual student. 

With the online homework being graded as students complete each problem, the instructor can 
monitor both individual and the class’s progress and success. One week between assigning the 
homework and its due date allowed the instructors to report the percentage of students starting the 
first problem to the class two to five days before the due date. Reporting the percent of students 
starting the homework during a class period served as a nice reminder to not wait until the last day 
to start, but this reporting process was not examined in detail. 
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Professor Mo dyn is fill ing a poorly insulated tank. Superheated steam is flowing through a supply line at 
2500 kPa anq435°C.jA valve connects the supply line to the tank, which has a volume of 1.000 m 3 . 

of 


t iall y, th e valve is erased and the tank contains a saturated liquid-vapor mixture of water with a pressure 
275 kPa and a quali ty of 50.2 %. The valve is opened and the steam is allowed to flow into the tank until 


a sa t u r a t ed vapor at|6000 kPalis obtained. At this point the valve is closed. Find the mass entering the tank 
and the heat interaction. Note. The IUPAC sign conversion for heat is used. Heat into the system has a 
positive value. 

Mass entering the tank 


Number 


□ 

kg 


Orange boxes indicate 3 rolling numbers in this problem. 
Two or three variations of T or P change the properties 
needed to solve the energy balance. 


Heat interaction 


Number 


□ 

kJ 


Problem statement with 
rolling numbers 


Hints help students 
start solving 

^ Hint 


Set up the energy balance and simplify. Is this 
process steady state? Is there an exit? 


Detailed explanation 
after incorrect response 


Simplify the general energy balance, 
remembering that integration from state one to 
state two needs to be done because it is not a 
steady state process. 

Use your steam tables to find the properties. 

You can find the mass in, the mass in the tank 
at state one, and the mass at state two by 
using the volume of the tank and the specific 
volume at each. The mass in is equal to the 
difference in the mass at each state. 

Make sure to use the quality to find the 
properties at state 1. 


Figure 1. Three screenshots of a tank filling problem from a thermodynamics course on 
Sapling Learning. The problem statement contains rolling numbers (top), hints (bottom left), 
and feedback for incorrect answers (bottom right). 


The Sapling grade system presents two qualitative assessments of the students’ success as well 
as quantitative data. First, a color-coded horizontal bar graph for each question can be viewed in 
the activity editor (Figure 2). Specifically, the size of the green bar correlates with the percent of 
students correctly solving the problem, orange signifies the percentage of students answering in¬ 
correctly, and white represents the fraction of the class that did not attempt the problem. Second, 
the grade book summarizes the success on each column with a single color (e.g., slightly different 
shades of green in Figure 2) as well as a colored box for each question and individual student. Quickly 
scanning the colored grid for the entire class can help an instructor determine individual or groups 
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# 

Stats 

ID 

Title 

Topic Description 

Difficulty 

1 [ 

i 

61625 

Double pipe counter flow heat exchanger 

Thermodynamics 

• #o 

2 1 

i 

69131 

Heat Exchanger - Water and Ethylene glycol 

Thermodynamics 

#+o 

3 l 

_l 

65294 

Pneumatic accumulator 

Thermodynamics 

##» 

4 [ 

i 

82709 

Tank filling with steam 

Thermodynamics 

»#» 


Figure 2. Screenshots of the assessment of homework problems in Sapling Learning. 
The activity editor shows progress of the class under stats (top). The grade book gives a 
color coded summary (middle leftX and the individual question statistics quantify correct, 
incorrect , and did not attempt (bottom). Two questions (65294 and 82709) appear in all 
three sections of the figure. 


Question Stats 

Item ID: 82709 
Title: Tank filling with steam 
Topic: Thermodynamics 
Difficulty: Hard 

Correct: 162 
Avg Score: 93.3% 

Avg Attempts: 0.93 

Incorrect: 26 
Avg Score: 37.1% 

Avg Attempts: 0.37 

Did not attempt: 21 


Activity editor 


Grade book 


Question Number 

1 

2 

3 

4 

Item ID 

61625 

69131 

65294 

82709 


Individual question statistics 


Question Stats 

Item ID: 65294 
Title: Pneumatic accumulator 
Topic: Thermodynamics 
Difficulty: Hard 

Correct: 188 
Avg Score: 94.6% 

Avg Attempts: 0.95 

Incorrect: 6 
Avg Score: 0% 

Avg Attempts: 0 

Did not attempt: 15 


of students who are attempting incorrectly compared to students who did not attempt some or all 
of the questions. The professional judgment of the instructors is needed to identify individual stu¬ 
dent’s struggles beyond the qualitative colors. Overall, scanning the grade book’s colors is singularly 
qualitative compared to the more descriptive multi-color bars from the activity editor. Following 
this qualitative observation, the quantitative data on the problems of interest can be examined. 

The quantitative data for online homework problems are the most informative (below right of 
Figure 2), especially for data-driven engineering professors. The Sapling grade system tallies the 
number of students earning the correct or incorrect answers for the entire problem and the per¬ 
centage earned by each group. For the two questions presented in the figure, a similar green color 
is observed in the grade book while the performance on problem 82709 is measurably lower than 
65294 when examining the quantitative question statistics. From using the Sapling system for sev¬ 
eral years 9 , the number of incorrect responses is the best indicator of a problem that the students 
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struggled with. Thus, using quantitation question statistics from the online homework, JITT exercises 
can focus on the most difficult concepts on recent course material, which will be discussed later. 


RESULTS 


Course logistics 

The Introduction to Thermodynamics course was delivered during the 2014 Fall semester at the 
Colorado School of Mines. The course number was CBEN210 so the material is considered sophomore 
undergraduate level. The major concepts covered in the course included: demonstrating a logical 
and rigorous problem solving ability, computing thermodynamic properties of pure fluids - such as 
water/steam, and solving steady state and transient processes using the first and second laws of 
thermodynamics. 

While 67% of the students were of sophomore standing when taking the course, junior (25%) and 
senior (8%) students also enrolled. The diversity in class standing is a result of the students’ different 
majors: 30% chemical engineering (ChE), 32% chemical and biochemical engineering (CBE), 20% 
engineering physics, 8% civil engineering, and 10% other engineering or science majors. The ChE 
and CBE students use this thermodynamics course as pre- or co-requisite for another sophomore 
course (material and energy balances) while the civil and physics students have less need for the 
course to complete their degrees. Some analysis comparing ChE/CBE students with the other majors 
was completed. Applying JITT affected all majors in a similar manner, so the results and discussion 
will focus on the JITT/No JITT comparison. 

The three authors delivered two sections of the course in parallel during the Fall 2014 semester. 
The large section enrolled 161 students with Professors Liberatore and Vestal co-teaching. Professor 
Morrish taught the small section with 51 students. The large section met Monday, Wednesday, and 
Friday from 8:00 to 8:50am while the small section met for 75 minutes on Tuesdays and Tuesdays 
starting at 12:30pm. The three instructors met weekly to keep the content delivered each week in 
sync. The average GPA for each section before the semester was calculated and found to be statis¬ 
tically equivalent (+/- 0.02 points). 

Both sections employed the same active learning strategies: short periods of lecture, usually 15 
minutes or less, were interspersed with examples. Generally, examples were worked in groups of 2 
or 3 with the instructor or instructors walking the room to answer questions and provide encourage¬ 
ment. The large section included 4 total instructors with two graduate student teaching assistants 
helping the two professors during most class periods. Thus, the student to instructor ratio was ~ 
40:1 in the large section and ~ 50:1 in the small section. Creating the small classroom within a large 
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classroom has been documented elsewhere 9 - 27 . In addition, a number of examples throughout the 
semester employed YouTube videos. The YouTube pedagogy has been published previously 28 ' 30 , and 
the YouTube problems were the same for both sections. Overall, the teaching styles and examples 
were similar between the two sections, and therefore, the presentation of Just-In-Time-Teaching 
examples could serve as a basis to compare the two sections. 

The instructors’ response using JITT took a number of forms. The JITT exercise is completed 
at the beginning of class, and the response includes some lecture related to the troublesome 
concept followed by active problem solving. Generally, the JITT problems were a variation of 
the homework problem with the lowest ‘score’ (problem response data was discussed earlier). 
Short answer or sketching diagrams were completed with neighboring students so the results 
could be discussed and refined. Longer problems, similar in length to a homework problem, were 
also given and took 10 to 15 minutes of the class period. Some of the problems were based on 
YouTube videos and problems written by students in previous semesters as published elsewhere 
28 ~ 30 . Overall, the amount of class time was budgeted relative to the success, or lack thereof, on 
the just completed homework. 

Student Achievement 

To illustrate the JITT technique, the steps and results pertaining to one course concept, tank filling, 
will be presented first and followed by the overall outcomes from the semester long application of 
JITT. Tank filling can be a challenging problem type for students due to the system being both open 
and transient. Each section of the course worked an example tank filling problem that applied the 1 st 
Law of Thermodynamics in class. Then the students were assigned two problems covering the topic 
on their homework. One of the tank filling questions was presented earlier in Figure 1. 

Tank filling using a real fluid (water) was identified as the most difficult problem on the homework 
set and provided the basis for the JITT example during the following class period. However, as a 
control, only the large section completed the JITT tank filling example; whereas the material was not 
reiterated in class for the small section. During the class period following the JITT exercise, all stu¬ 
dents were given a quiz covering a 1 st law energy balance on a tank filling process. The performance 
for the two sections on the quiz is shown in Figure 3 with the JITT section clearly out scoring the 
section not receiving JITT by over 25%. Some disparity between the two average scores could be 
due to a difference in the actual quiz problem. Since the quiz was given on two separate days, the 
question was slightly different between the two sections to discourage cheating. Both versions of 
the quiz awarded 8/10 points for the exact same energy and mass balance formulas, but the large 
section’s tank filling problem used water as the fluid while the small section’s quiz used an ideal 
gas as the fluid. While the variation in quiz content could account for up to 20% of the difference 
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0 

Q. 



Assessment Type 


Figure 3. Average performance on tank filling assessment for both a quiz and a final exam 
problem. The different bar shading represents sections that were given JiTT and those that 
were not (No JiTT). The error bars represent 95% confidence intervals. 


between JITT and non-JITT students, the data would still show a statistically significant performance 
improvement when utilizing a JITT review. 

To eliminate variability between sections as a differentiating factor, the same experiment was 
conducted a second time just before the final exam with only the smaller section given a JITT exer¬ 
cise on tank filling. There was a significant span of 7 weeks between the quiz and final exam. Again, 
the group that received the JITT review scored higher than the section without the JITT feedback. 
In this case, the final exam problem was exactly the same for all students (all exam questions con¬ 
tained the same problem statements with three different sets of numbers to dramatically reduce 
the chance of cheating). Certainly, it is intuitive that providing students with extra practice on a 
topic can improve understanding. However, the added benefit of the JITT approach is in identify¬ 
ing and strategically targeting the most challenging concepts. We believe that extra time spent in 
class reiterating this identified material will pay dividends on student achievement. These results 
given in Figure 3 substantiate that JITT methodology can provide measureable increases in student 
performance on a topic. 

Now, the application of JITT throughout the semester and its impact on student achievement will 
be examined. In this case, the topics were presented similarly in both sections. Table 1 shows a list 
of the five topics covered using JITT and the five topics covered without using JITT. Because this 
study was conducted with a single group of students, the topics could not be the exact same for 
both JITT and No JITT. However, we specifically chose concepts spanning the entire semester and 
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Table 7 . Comparison of Sapling homework average to the average on a similar quiz or 
exam problem for JITT and No JiTT topics. The ± values represents the 95% confidence 
interval for exam/quiz scores. Standard deviations are not tabulated in the Sapling 
platform and consequently confidence intervals for HW% could not be calculated. 


Topic 

HW% 

quiz/exam% 

JITT 



Gauge pressure, fluid head 

74.3 

65.0 ±3.5 

Steam quality/interpolation 

82.9 

71.2 ±3.4 

1st Law piston/cylinder two-step 

72.9 

77.6 ± 3.4 

1st Law nozzle 

77.8 

85.0 ±2.3 

1st Law turbine 

74.3 

87.6 ± 2.4 

No JITT 



Ideal Gas Law and unit conversion 

77.2 

80.0 ± 3.4 

1st Law piston/cyUnder single-step 

87.8 

87.0 ±2.1 

1st Law rigid tank 

88.5 

68.0 ±4.4 

Carnot refrigerator 

83.7 

89.0 ± 2.3 

Closed, reversible piston/cylinder 

79.4 

66.0 ± 3.6 


which we believe are representative of the full course. Although some variability due to differing 
topics is unavoidable, in total the overall data represents nearly 2000 data points making compari¬ 
sons between the two cases reasonable. 

Figure 4 below shows summary statistics of performance on exams and quizzes for three sce¬ 
narios: a) topics covered by JITT b) topics not covered by JITT and c) overall comparisons of the two. 
For the box plots, the middle line of the box represents the median score with the bottom and top 
edges showing the first and third quartiles, respectively. For example, a third quartile score of 85% 
indicates that 75% of the students scored 85% or less on the quiz or exam problem. The whiskers 
display the maximum and minimum scores achieved. Each box represents all students that were 
assessed, typically 180 or more. 

In all instances, the top quartile whisker is shorter than the bottom quartile whisker indicating 
score distributions were not normal. This result is expected given a forced maximum of 100%. Figure 
4c provides side-by-side comparisons of overall performance on all topics covered by JITT versus 
No JITT. The mean quiz/exam scores were nearly identical at 77.5 ± 2.5% for JITT and 78.1 ± 2.9% for 
No JITT. However, the JITT data did show a higher median value of 83.1% compared to 80.8% for no 
JITT. The 3 rd quartile was 95.5% for JITT and 93.0% for No JITT whereas the 1 st quartile values were 
nearly equivalent at 67.3% and 67.8%, respectively. 
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Figure 4. Summary exam/quiz performance statistics for a) topics covered by JITT b) topics 
not covered by JITT (No JITT) and c) Overall statistical comparisons between the two. The 
error bars represent 95% confidence intervals. 


This data suggests that while average performance for all students was not impacted by JITT, 
inclusion of the technique could help shift the mid-performing students upward as indicated by the 
increased median and 3 rd quartile scores. While individual students were not tracked to see if JITT 
shifted students to a higher quartile on any single assignment or over the course of the semester, 
the average performance provides an aggregated comparison over a sizable number of students. 
One possible explanation is that these mid-performing students gain more benefit from repetition 
of challenging material via JITT than the high or low performing students. This result could translate 
to more B’s and fewer C’s in a course. 
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The exam/quiz performance box plots in Figure 4a and b are provided as a function of the Sap¬ 
ling homework score on the corresponding topic, though in this work no trend between the two 
was observed. In general, a topic that is challenging on the homework would also be so during as¬ 
sessment. But given a data set of only ten topics, this relationship is blurred by varying degrees of 
difficulty of the exam and quiz problems. Trends between assessment and homework performance 
could emerge with a larger number of topics to compare. 


Student Surveys 

An end of semester survey collected students’ opinions on various aspects of the course. Four 


questions were asked about the JITT exercises (Figure 5). Over 65% of the class thought the amount 
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Figure 5. Student responses to survey questions (n=183 students), a) the amount of class 
time used for JITT versus three choices, b) percentage of students agreeing or strongly 
agreeing with three statements related to JITT. 


SPRING 2017 


11 




























ADVANCES IN ENGINEERING EDUCATION 

Effectiveness of Just In Time Teaching on Student Achievement in an 

Introductory Thermodynamics Course 


of class time devoted to JITT was appropriate with a small fraction wanting more feedback (-11%). 
The other three questions returned overwhelming positive response for: 1. Feeling like an active 
participant in the class, 2. Effectively using class time, and 3. Liking the review of the most difficult 
homework concepts. Overall, the survey results are consistent with other studies 7>9 , and JITT pro¬ 
vides an opportunity for engagement. Engagement using technology is a widely studied topic 15 - 31 
and the results presented here add to this growing dialogue. 


CONCLUSIONS 

An approach using online homework to guide Just-In-Time-Teaching exercises in an Introduc¬ 
tion to Engineering Thermodynamics course found differences in student performance. The online 
homework platform provided immediate feedback to students and revealed problematic concepts 
for the class as a whole. Once identified, these challenging topics were reiterated during the class 
period following homework submission. The impact of using the JITT exercises on student achieve¬ 
ment was then evaluated using in class quizzes and exams. With about 200 students in the course, 
student performance on a topic covered by a JITT exercise was found to be measurably higher 
(> 9.0%) than a control group who did not receive the next-day review. While less definitive, com¬ 
parison of student achievement on a series of topics covered by JITT versus those not covered (No 
JITT) did show positive outcomes of the approach. The average median and 3 rd quartile scores were 
higher for JITT topics. These results may indicate that the immediate topical review is more helpful 
for the mid-performing students. Overall, student feedback on JITT exercises was highly favorable 
with over 85% of students responding that JITT exercises were helpful for engagement and a good 
use of class time. 
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